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ABSTRACT 


In this study we investigate the abilities to determine credibility of digital news among 532 teenagers. Using an online 
test we assess to what extent teenagers are able to determine the credibility of different sources, evaluate credible and 
biased uses of evidence, and corroborate information. Many respondents fail to identify the credibility of false, biased and 
vetted news. We identify a digital divide between people with and without the ability to determine credibility. We also 
find that a large proportion of the respondents struggle to identify the source of information in Sweden’s most read online 
newspaper. Respondents struggle to determine the bias of news reports regarding racism and weight loss, but are better at 
debunking manipulated images. Respondents who value the importance of credible news and who indicate that they have 
learned media literacy in school seem to hold a mind-set helping them to determine credibility better than other 
respondents. Our findings provide a basis for further research of how to better understand and support digital civic 
literacy in classrooms and society. 
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1. INTRODUCTION 


New media may connect people across cultural and ideological borders, but also foster mistrust. Digital news 
media is pivotal to stimulate conversations and active citizenry (e.g., Lee, Shah, & McLeod, 2013). However, 
digital media and modern journalism can also in many ways facilitate the spread of exaggerations, rumours 
and lies (e.g., Vosoughi, Roy, & Aral, 2018). Today mainstream and non-mainstream news are mixed in 
digital environments making it difficult to assess credibility (Fletcher & Park, 2017). This implies new 
requirements for both readers and society (e.g., McGrew et al, 2017, 2018). Scholars argue that teaching and 
learning what we label digital civic literacy is absolutely essential to informed and engaged citizenship 
(e.g. Kahne & Bowyer, 2017). We define digital civic literacy as the ability to navigate digital civic 
information in critical and constructive ways, and we acknowledge that simply having access to information 
is not enough. Digital civic literacy enable citizens to engage with political and social topics in critical and 
constructive ways, but it has proven to be quite a challenge to implement digital tools and digital literacy in 
education (Balanskat et al. 2013). Previous research highlights how prior beliefs, coherence of the message, 
and cognitive ability may affect people’s ability to assess the credibility of information (Lewadowsky et al., 
2012). It has also been noted how attitudes to news, scientific curiosity and socio-economic factors may 
influence how people navigate new information, facts and digital environments (e.g., Strémbick, 
Djerf-Pierre, & Shehata, 2013; Kahan et al., 2017; Hatlevik et al., 2015). People use cues and heuristics to 
estimate credibility, which may be helpful, but they may still fail to separate false and biased news from 
authentic information regarding debated topics (e.g., Wineburg & McGrew, 2018). 

The aim of the current study was to investigate youths’ ability to assess the credibility of information on 
the Internet and how this relates to personal characteristics and beliefs, which in turn will enable us to 
suggest educational measures to support and scaffold young people’s abilities to assess online information. 
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2. METHOD AND DESIGN 


Researchers in education and psychology designed and piloted, in close collaboration with in-service 
teachers, online multiple choice tests in order to assess and compare youths (NV = 532) age 15-19. The tests 
consisted of 22 questions whose contents and measured variables are presented in Table 1. The design of the 
tests was inspired by previous research on civic online reasoning (McGrew et al., 2017). In the present test, 
the participants were also asked for their self-rated ability to find news online, critical ability, the importance 
of credibility of news, how reliable information on the Internet is, and how much media literacy education 
they have had in school. All self-rated variables were on a 5 point scale, with the exception of media literacy, 
which was on a 10 point scale. Finally, demographic variables and media habits were measured (see Table 1 
for a more detailed account). 


Table 1. Overview of questions and measured constructs in the online test. The measured constructs refer to abilities that 
have been identified as important for the detecting of false news online. The questions are referred to in the text with the 
names in parentheses 


Measured construct Question 


Detecting sponsored material (sourcing) Screenshot from IT journal (Techworld) 
Screenshot from evening paper 1 (Aftonbladet) 
Screenshot from evening paper 2 (Expressen) 


Comparing articles (corroboration) Two articles about weight loss (weight loss) 
Two articles about the government’s policy on racism (racism) 


Scrutinising comments and images (evidence) Edited photograph of a smoking girl (smoking) 
Article about the government’s energy goals (energy goals) 
Article about the climate change (climate change) 
Reader’s comment on incomes (income) 
Edited photograph on daisies in Fukushima (Fukushima) 


Self-rated abilities Media literacy education (media literacy) 
Reliability of information on the Internet (information reliability) 
Source criticism on the Internet (fact-checking ability) 
Finding information on the Internet (critical ability) 
Importance of news credibility (credibility importance) 


Background variables Gender 
Form (youths)/Age group (adults) 
Secondary school focus (youths) 
Political party sympathies (1 or more parties) 
News format (e.g., paper news, radio etc.) 
News source (e.g., local newspaper, evening paper etc.) 


3. RESULTS 


We first computed the total number of correct answers for all question items that could be classified as 
correct or incorrect, where the questions with embedded advertisements (see Table 1) was coded as correct if 
all answers were correct. The mean number of correct of maximum 8 for each for each form were: ninth form 
compulsory (M = 3.38, SD = 1.72), first form secondary (M = 3.42, SD = 1.74), second form secondary 
(M = 3.62, SD = 1.74), and third form secondary (M = 3.48, SD = 1.82). The items that the pupils found most 
difficult were Aftonbladet (12% correct), Expressen (21 % correct), Techworld (34% correct), racism (43%) 
and weight loss (53%). 

This should be contrasted with the mean credibility ratings for the articles: climate change 
(M = 6.02, SD = 2.26), energy goals (M = 4.69, SD = 2.13), and income (M = 4.05, SD = 1.87). Hence, there 
were no clear differences between the articles, notably, the false news article (climate change) was rated as 


376 


15th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2018) 


the most credible article. The self-rated abilities were: fact-checking ability (VM = 3.99, SD = 0.743), 
credibility importance (M = 3.95, SD = 1.12), critical ability (MW = 3.99, SD = 0.743), media literacy 
(M = 7.18, SD = 2.07), and information reliability (M = 2.87, SD = 0.755). 

In order to relate performance to background variables and self-rated abilities we performed a number of 
regressions. Results showed that for Expressen, speaking Swedish at home and high ratings on media literacy 
were associated with an increased probability of correct answers, whereas a higher self-reported rating on 
fact-checking ability was associated with a decreased probability of number of correct items. For 
Aftonbladet, a higher rating on credibility importance was associated with an increased probability of number 
of correct items and for Techworld, speaking Swedish at home was associated with an increased probability 
of number of correct items. 
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Figure 1. Panel 1 depicts the predicted probabilities of a correct answer on the comment on incomes as a function of 
information reliability (x-axis) and fact-checking ability (legend). The higher ratings on both variables, the smaller 
probability of a correct answer. Panel 2 depicts the predicted probabilities of a correct answer for smoking as a function 
of media literacy and credibility importance. The higher ratings on both variables, the higher probability of a correct 
answer 


For smoking, a higher rating on media literacy, information reliability, and credibility importance were 
associated with an increased probability of a correct answer, whereas a higher rating on fact-checking ability 
was associated with a decreased probability of a correct answer (illustrated in Figure 1). For income, a higher 
rating on media literacy and credibility importance were associated with an increased probability of a correct 
answer, whereas a higher rating on information reliability and fact-checking ability were associated with a 
decreased probability of a correct answer (illustrated in Figure 1). For racism, a higher rating on media 
literacy in school was associated with an increased probability of a correct answer, whereas a higher rating on 
media literacy, information reliability and credibility importance were associated with a decreased 
probability of a correct answer. Finally, for Fukushima, a higher rating on media literacy and credibility 
importance were associated with an increased probability of a correct answer, whereas a higher rating on 
fact-checking ability and information reliability were associated with a decreased probability of correct 
answer. 

To sum up, a higher rating on credibility importance and media literacy in education were associated with 
an increased probability of correct answers. The inverse is true for self-reported fact-checking ability and 
information reliability — here a higher rating was associated with a decreased probability for a correct answer. 
The picture of a smoking girl and the articles on racism, however, do not follow this pattern: for the smoking 
girl a higher rating of the reliability of information is associated with an increased probability of a correct 
answer, and for the articles on racism a higher rating on credibility importance is associated with a decreased 
probability of a correct answer. Further investigation is needed to understand why the results for these 
models differ from the others. Finally, speaking Swedish at home was associated with a higher probability for 
a correct answer on more than half of the items. 
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4. CONCLUDING DISCUSSION 


In this short paper we present three preliminary main results: the performance on all items were relatively 
poor, less than half of the items were correct; second, the credibility ratings on three articles, one which was 
fake news, were relatively high and did not differ; finally, regressions with number of correct item/correct 
answer as outcome variables showed that self-rated importance of credibility of news and media literacy in 
education were associated with an increased probability of a correct answer, whereas a self-reported higher 
rating on fact-checking ability and reliability of news on the Internet were associated with a decreased 
probability of a correct answer. This is indicative of the existence of subgroups in the population, which in 
turn implies that there is a digital divide between people, who are skilled at determining credibility and others 
who are not. In one group we find people who rate credibility as very important who do not claim to be vary 
skilled at fact-checking and finding information online. In the opposite group we find people who do not find 
it very important with reliable information and see themselves as good critical readers. At present, we do not 
know what characterises individuals with low digital civic literacy skills and problematic high self-ratings of 
fact-checking ability and reliability of information on the Internet. A speculative account is that this may be 
interpreted as a mind-set of ignorance enhancing confirmation bias. People doing well on our test may in 
contrast have a mind-set with scientific curiosity and an openness to consider biases, also their own (Kahan 
et. al. 2017). Perhaps our findings also indicate a gap between news-seekers and news-avoiders (Strémbick, 
Djerf-Pierre, & Shehata, 2013). 

Teaching and learning media literacy seem to be important. Our findings indicate also that digital civic 
literacy may be linked to appreciating the importance of reliable information and understanding how it is 
difficult to find and evaluate online information. We need to further investigate this in education and how 
education may support media literacy and a more humble and curious approach to information. 
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